[FLOC-4489] Use detailed volume list and update to a version of mimic that supports that endpoint #2892

wallrj · 2016-08-30T15:51:58Z

Fixes: https://clusterhq.atlassian.net/browse/FLOC-4489
Depends on: ClusterHQ/mimic#1

I re-instated the use of detailed volume list.
This ensures that volume attachment information is returned, even when interacting with Cinder v2 API.

One of our tests attempts to hit the volumes/detail endpoint with a mimic based fake REST server.
So I also had to fork andupdate mimic to support that endpoint.

I'll post devstack based test results shortly.

wallrj · 2016-08-30T17:09:22Z

I'm trying to understand some separate failures on devstack


Traceback (most recent call last):
  File "/home/ubuntu/flocker/flocker/node/agents/functional/test_cinder.py", line 444, in test_get_device_path_no_attached_disks
    device_path = self.blockdevice_api.get_device_path(volume.id)
  File "/home/ubuntu/flocker/flocker/node/agents/cinder.py", line 726, in get_device_path
    device_path = self._get_device_path_virtio_blk(cinder_volume)
  File "/home/ubuntu/flocker/flocker/node/agents/cinder.py", line 678, in _get_device_path_virtio_blk
    raise UnattachedVolume(volume.id)
flocker.node.agents.blockdevice.UnattachedVolume: b0c589bd-20b6-429d-9686-936352775c1b


flocker.node.agents.functional.test_cinder.CinderAttachmentTests.test_get_device_path_no_attached_disks
-------------------------------------------------------------------------------
Ran 1 tests in 15.566s

…o cinder-volume-detail-FLOC-4489

…' into cinder-volume-detail-FLOC-4489

…pdate the cleanup

wallrj · 2016-09-02T14:37:46Z

I'm trying to understand some separate failures on devstack

These seem to occur only on Ubuntu 14.04 and the problem is that the hotplug system doesn't always create a /dev/disks/by-id symlink for the device that's been attached.

I can force the symlink by running udevadm trigger

The tests are much more reliable on Ubuntu 16.04

wallrj · 2016-09-02T14:40:59Z

(4489) ubuntu@richardw-xenial-machine-1:~/flocker$ sudo /usr/bin/env FLOCKER_FUNCTIONAL_TEST_CLOUD_CONFIG_SECTION=storage-drivers.devstack FLOCKER_FUNCTIONAL_TEST_CLOUD_CONFIG_FILE=/home/ubuntu/acceptance.yml FLOCKER_FUNCTIONAL_TEST=TRUE $(type -p trial) flocker.node.agents.functional.test_cinder
sudo: unable to resolve host richardw-xenial-machine-1
flocker.node.agents.functional.test_cinder
  BlockDeviceAPIDestroyTests
    test_destroy_timesout ...                                              [OK]
  CinderAttachmentTests
    test_get_device_path_no_attached_disks ...                             [OK]
  CinderBlockDeviceAPIInterfaceTests
    test_attach_attached_volume ...                                        [OK]
    test_attach_destroyed_volume ...                                       [OK]
    test_attach_elsewhere_attached_volume ...                              [OK]
    test_attach_unattached_volume ...                                      [OK]
    test_attach_unknown_volume ...                                         [OK]
    test_attached_volume_listed ...                                        [OK]
    test_compute_instance_id_nonempty ...                                  [OK]
    test_compute_instance_id_unicode ...                                   [OK]
    test_created_is_listed ...                                             [OK]
    test_created_volume_attributes ...                                     [OK]
    test_destroy_destroyed_volume ...                                      [OK]
    test_destroy_unknown_volume ...                                        [OK]
    test_destroy_volume ...                                                [OK]
    test_detach_detached_volume ...                                        [OK]
    test_detach_unknown_volume ...                                         [OK]
    test_detach_volume ...                                                 [OK]
    test_device_size ...                                                   [OK]
    test_foreign_cluster_volume ...                                        [OK]
    test_foreign_volume ...                                                [OK]
    test_get_device_path_device ...                                        [OK]
    test_get_device_path_device_repeatable_results ...                     [OK]
    test_get_device_path_unattached_volume ...                             [OK]
    test_get_device_path_unknown_volume ...                                [OK]
    test_interface ...                                                     [OK]
    test_list_attached_and_unattached ...                                  [OK]
    test_list_volume_empty ...                                             [OK]
    test_listed_volume_attributes ...                                      [OK]
    test_multiple_volumes_attached_to_host ...                             [OK]
    test_name ...                                                          [OK]
    test_reattach_detached_volume ...                                      [OK]
  CinderCloudAPIInterfaceTests
    test_current_machine_is_live ...                                       [OK]
    test_interface ...                                                     [OK]
    test_list_live_nodes ...                                               [OK]
  CinderFromConfigurationTests
    test_no_immediate_authentication ...                                   [OK]
    test_no_retry_authentication ...                                       [OK]
  CinderHttpsTests
    test_verify_ca_path_no_match_fails ...                            [SKIPPED]
    test_verify_false ...                                             [SKIPPED]
  LazyLoadingProxyTests
    test_loader_called_once ...                                            [OK]
    test_loader_exceptions_raised ...                                      [OK]
  VirtIOCinderAttachmentTests
    test_disk_attachment_fails_with_conflicting_disk ...              [SKIPPED]
    test_get_device_path_correct_with_attached_disk ...               [SKIPPED]
    test_get_device_path_virtio_blk_error_without_udev ...            [SKIPPED]
    test_get_device_path_virtio_blk_symlink ...                       [SKIPPED]

===============================================================================
[SKIPPED]
Tests require a TLS auth_url endpoint beginning with https://. Found auth_url: http://172.99.85.118:5000/v2.0

flocker.node.agents.functional.test_cinder.CinderHttpsTests.test_verify_ca_path_no_match_fails
flocker.node.agents.functional.test_cinder.CinderHttpsTests.test_verify_false
===============================================================================
[SKIPPED]
Tests require the ``virsh`` command.

flocker.node.agents.functional.test_cinder.VirtIOCinderAttachmentTests.test_disk_attachment_fails_with_conflicting_disk
flocker.node.agents.functional.test_cinder.VirtIOCinderAttachmentTests.test_get_device_path_correct_with_attached_disk
flocker.node.agents.functional.test_cinder.VirtIOCinderAttachmentTests.test_get_device_path_virtio_blk_error_without_udev
flocker.node.agents.functional.test_cinder.VirtIOCinderAttachmentTests.test_get_device_path_virtio_blk_symlink
-------------------------------------------------------------------------------
Ran 45 tests in 401.702s

PASSED (skips=6, successes=39)

wallnerryan · 2016-09-06T14:21:23Z

flocker/node/agents/cinder.py

@@ -530,7 +530,7 @@ def list_volumes(self):
        http://docs.rackspace.com/cbs/api/v1.0/cbs-devguide/content/GET_getVolumesDetail_v1__tenant_id__volumes_detail_volumes.html
        """
        flocker_volumes = []
-        for cinder_volume in self.cinder_volume_manager.list(detailed=False):
+        for cinder_volume in self.cinder_volume_manager.list(detailed=True):


Is there a case where we do not always need detailed=True? I seem to remember the discussion that Cinder v1 would pass with detailed=False, thus the detailed information is not needed? maybe im making this up, but making sure we're not requested uneeded information that may slow down the process being that every volume may have more information.

Yes that's right.

Cinder v1 API volume list includes metadata: http://developer.openstack.org/api-ref-blockstorage-v1.html#listVolumes

Cinder v2 doesn't: http://developer.openstack.org/api-ref-blockstorage-v2.html#listVolumes

That's what led to this bug. I tested against Rackspace and saw the tests pass.
On OpenStack the detailed list doesn't include the attachment info.

Should it be the case where we use False for v1 instead of always True for detailed?

It'd be possible but probably more trouble than it's worth.
At this point the cinder version detection has already taken place and the CinderBlockDeviceAPI has an object conforming to an interface that we've defined ICinderVolumeManager.
Since we're aiming to drop support for CinderV1 (Rackspace) I'm going to leave it.

ok. I wonder if we should drop v1 though, there may be private deployments that may want to use v1, but not sure.

wallnerryan · 2016-09-06T17:58:25Z

Did a preliminary review, but will revisit when ready and change status in JIRA, i think because this was at the top of a stack of other changes. Also looks like 1 check failing, looks like the same failure happening on #2893

…l-FLOC-4489

wallnerryan · 2016-09-07T12:37:48Z

@wallrj
You mentioned...

I can force the symlink by running udevadm trigger

to get /dev/disks/by-id to show up. Is this something that needs to be fixed for 14.04, this sounds like a bug to me.

Also, stilling seeing the following error in the trusty cinder CI test

Traceback (most recent call last):
  File "/tmp/14124/local/lib/python2.7/site-packages/testtools/twistedsupport/_runtest.py", line 386, in _log_user_exception
    raise e
testtools.twistedsupport._spinner.TimeoutError: <flocker.acceptance.endtoend.test_diagnostics.DiagnosticsTests.test_export id=0x7f4c4d752b90> took longer than 300.0 seconds

]
test: flocker.acceptance.endtoend.test_dockerplugin.DockerPluginTests.test_create_container_with_v2_plugin_api
Build timed out (after 90 minutes). Marking the build as failed.
Build was aborted
Archiving artifacts
Archiving artifacts

If these concerns can safely be ignored or are unrelated then the PR looks good. Please answer the above but otherwise LGTM

wallrj · 2016-09-08T09:54:40Z

Thanks @wallnerryan

to get /dev/disks/by-id to show up. Is this something that needs to be fixed for 14.04, this sounds like a bug to me.

Yeah, we do need to investigate that.

It only affects virtio OpenStack environments...which is where we do disk serial number based device path detection.

It shouldn't affect Rackspace, which uses Xen volume driver.

So I'll merge this and create followup issues:

hotplug failure: https://clusterhq.atlassian.net/browse/FLOC-4493
Rackspace Ubuntu 14.04 acceptance test timeout: https://clusterhq.atlassian.net/browse/FLOC-4494

Install ClusterHQ fork of mimic

cbf4eb2

wallrj added 3 commits September 2, 2016 12:25

Merge remote-tracking branch 'origin/flaky-mimic-tests-FLOC-4491' int…

c3718cd

…o cinder-volume-detail-FLOC-4489

Merge remote-tracking branch 'origin/rackspace-libcloud-tls-FLOC-4490…

764a226

…' into cinder-volume-detail-FLOC-4489

The updated mimic is detected as a cinder v2 endpoint so we need to u…

84bb3b7

…pdate the cleanup

wallnerryan reviewed Sep 6, 2016
View reviewed changes

Merge remote-tracking branch 'origin/master' into cinder-volume-detai…

fe8f1a3

…l-FLOC-4489

wallrj merged commit 726abc3 into master Sep 8, 2016

wallrj deleted the cinder-volume-detail-FLOC-4489 branch September 8, 2016 09:55

wallrj mentioned this pull request Sep 20, 2016

openstack cinder volume metadata AttributeError #2889

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLOC-4489] Use detailed volume list and update to a version of mimic that supports that endpoint #2892

[FLOC-4489] Use detailed volume list and update to a version of mimic that supports that endpoint #2892

wallrj commented Aug 30, 2016

wallrj commented Aug 30, 2016

wallrj commented Sep 2, 2016

wallrj commented Sep 2, 2016

wallnerryan Sep 6, 2016 •

edited

Loading

wallrj Sep 6, 2016

wallnerryan Sep 6, 2016

wallrj Sep 7, 2016

wallnerryan Sep 7, 2016

wallnerryan commented Sep 6, 2016 •

edited

Loading

wallnerryan commented Sep 7, 2016 •

edited

Loading

wallrj commented Sep 8, 2016

[FLOC-4489] Use detailed volume list and update to a version of mimic that supports that endpoint #2892

[FLOC-4489] Use detailed volume list and update to a version of mimic that supports that endpoint #2892

Conversation

wallrj commented Aug 30, 2016

wallrj commented Aug 30, 2016

wallrj commented Sep 2, 2016

wallrj commented Sep 2, 2016

wallnerryan Sep 6, 2016 • edited Loading

Choose a reason for hiding this comment

wallrj Sep 6, 2016

Choose a reason for hiding this comment

wallnerryan Sep 6, 2016

Choose a reason for hiding this comment

wallrj Sep 7, 2016

Choose a reason for hiding this comment

wallnerryan Sep 7, 2016

Choose a reason for hiding this comment

wallnerryan commented Sep 6, 2016 • edited Loading

wallnerryan commented Sep 7, 2016 • edited Loading

wallrj commented Sep 8, 2016

wallnerryan Sep 6, 2016 •

edited

Loading

wallnerryan commented Sep 6, 2016 •

edited

Loading

wallnerryan commented Sep 7, 2016 •

edited

Loading